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The aim of the study was to assess whether the Aggression Replacement Training (ART) 
programme is effective in increasing social skills and decreasing problem behaviour. The 
sample consisted of 232 children (mean age 10.9 yrs, SD = 2.32), their parents and 
teachers. The study had a quasi-experimental design with intervention and control 
groups. Children were recruited from six schools and four social institutions from four 
regions in North-West Russia from 2010 to 2013. Social skills and externalizing 
behaviour were assessed with the Social Skills Rating Scale and analyzed by repeated 
measures ANOVA (GLM). In a pre and post-test assessment, the 30-hour ART 
programme was associated with a significant increase in social skills when assessed by 
children’s self-reports. The most reliable effects of the intervention were demonstrated in 
the two age groups of 6-9 and 10-14 years old. When both pre and post-test were 
assessed by parents and teachers, children from both the intervention and control groups 
demonstrated more social skills and less problem behaviour. Overall results point to a 
significant improvement of social skills among children from the intervention groups, 
but an improvement in social skills and reduction of problem behaviour have also been 
indicated among children from the control group. Findings are discussed in view to 
possible diffusion of treatment from children participating in an intervention to children 
from control groups. 
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Introduction 

Children and young people with behavioural problems constitute between one third and a half of all 
referrals to support services in the USA (Kazdin & De Los Reyes, 2009). High prevalence of behavioural 
problems is also the case for the Russian Federation where from 15 to 20% of children were reported as 
having serious mental health problems according to Goodman, Slobodskaya & Knyazev (2005). Even if the 
population of children and young people with behaviour problems differs from country to country, overall it 
is a cause of concern both for the children themselves and for their surroundings. For example, in Norway, 7- 
12 % of all children aged 10-17 demonstrate such a high degree of undesirable behaviours that can be 
considered as behavioural problems. Of these, about 2% have severe antisocial behaviour (Nordahl, Sprlie, 
Manger & Tveit, 2005). According to the largest Scandinavian independent research organisation SINTEF, 
40% of children and young people referred to the psychiatric services in Norway, also have behavioural 
problems (SINTEF, 2004). As for the Russian Federation, in a study by Vermeiren, Deboutte, Ruchkin & 
Schwab-Stone (2002) it was found that 69% of the studied adolescents from the general population were 
reported as manifesting from moderate to severe antisocial behaviour. 

Among the factors that may prevent the development of behavioural problems is the ability to 
generate pro-social behaviour in general and especially in stressful situations. Thus, intervention programmes, 
which aim at increasing social competence, also have been found to decrease problem behaviour (Sprlie, 
2000). As social competence involves a number of different aspects including developing empathy, 
cooperation, self-control and assertiveness, programmes with multiple focuses and different modules seem to 
be more effective and successful than programmes orientated only towards one aspect of the problems 
(Dowden & Andrews, 2000). 

Among such multi-focused programmes aimed to increase social competence is Aggression 
Replacement Training (ART) (Glick & Gibbs, 2011; Goldstein, Glick & Gibbs, 1998). The programme 
consists of three equal components: social skills training, anger control training and moral reasoning training. 
Each component is taught on a weekly basis (3 sessions per week) over a 10-week period. Social skills 
training is the behavioural component in which participants learn how they ought to behave in social 
situations. Anger control training is the emotional component where participants learn strategies to manage 
anger. Moral reasoning training is concerned with cognitive behaviour and moral values where participants 
learn to take perspective of others. The programme has a fixed structure and makes considerable use of role- 
playing and exercises. There are also various strategies for the transfer and maintenance of the skills that have 
been developed. In the present study, character education was included in the moral reasoning component and 
the concept of setting events was incorporated in the anger control training component (Gundersen, Olsen & 
Finne, 2008). In addition, rehearsals and selected pedagogical techniques described by Gundersen and 
Moynahan (2006) were included. The programme can thus be seen as an extended version of ART. 

Several studies have documented the empirical efficacy of the ART programme both with children 
and adolescents (Currie, Wood, Williams, & Bates, 2012; Goldstein & Glick, 1994; Nugent & Bruley, 1998) 
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and with incarcerated youths (Bamoski & Aos, 2004). In Norway outcome effects have been positive 
(Gundersen & Svartdal, 2006, 2010; Langeveld, Gundersen, & Svartdal, 2012). Some of these evaluations 
have observed diffusion of treatment effect, i.e., that interventions intended for participants in the treatment 
group have also affected participants in the control group (Gundersen & Svartdal, 2010; Kazdin, 1998) 
threatening conclusions about treatment efficacy. This problem is often an experimenter-related issue because 
it is an effect of the improper implementation of the intervention (e.g. Kazdin, 1998). However, treatment 
diffusion can occur even if the implementation is properly executed, such as when the intervention changes 
children’s behaviour in the intervention group and then indirectly affects members of the control group. Such 
“secondary diffusion” is particularly likely when interventions are implemented in situations with a high 
degree of interaction between participants in intervention and control groups, as is the case in schools and 
institutions involved in the present study. 

This paper presents the results of an evaluation of the ART programme in north-western Russia. ART 
has been implemented in north-western Russian schools and institutions since 2009, but it has not yet been 
evaluated. Programme implementation runs on the lines of “Children and Youth at Risk” in the Barents 
region 2008 - 2015, a cooperation programme within the framework of the Barents Euro-Arctic Council. The 
overall goal of the study was to examine the efficacy of ART in relation to pro-social skill acquisition and 
decrease of externalizing problems. We expected significant changes in skills acquisition and decrease of 
externalizing problems among children in the intervention group when compared to children in the control 
group. Due to close day-to-day communication between the participants in the study, possible diffusion of 
treatment effect from the intervention group and hence corresponding changes in skills acquisition and some 
decrease of externalizing problems among participants in the control group could be expected. 

Methods and procedure 

Participants 

The participants in the study were children from four social institutions and six elementary schools 
located in the north-western part of Russia. A total of 232 children participated, 145 (44% girls) in the 
intervention and 90 (63% girls) in the control group. The mean age of the ART group was 10.6 yrs (SD=.20) 
and 11.3 yrs (SD=.23) for the control group. The study also involved teachers (41 in the ART group and 48 in 
the control group) and parents (29 in the ART group and 41 in the control group). The response rate in the 
study was very high with only 3 participants (2 in the ART and 1 in the control group) not completing the 
questionnaires at post-test. 

ART trainers and intervention 

The ART intervention was provided by teachers from related institutions and schools, trained by 
authorised programme trainers. Before the intervention, they received 72 hours of educational training, spread 
over three meetings. In addition, they had to accomplish 18 training sessions with colleagues before training 
took place with the youth groups. Children from the intervention group received the extended ART course, 
including 30 sessions in each of the three components spread over ten weeks. Each session lasted from 45 to 

ISSN 2073-7629 

© 2014 CRES/ENSEC Volume 6, Number 1, April 2014 pp 16 



90 minutes and was led by two qualified ART trainers. The number of children in the intervention and control 
groups varied from 4 to 6. Children in the control groups did not receive any other intervention at the time of 
the study other than the standard school or institution curriculum. 

The ART trainers’ responsibility was to provide the intervention with the children and they were not 
involved in completing the questionnaires. Children received a detailed description of the study and were 
informed about the voluntary and confidential nature of their involvement with the study. They were also 
assured that neither parents nor school or institution staff would obtain any individualized information about 
their responses. 

Assessment 

The Social Skills Rating System (SSRS) (Gresham & Elliot, 1990) was selected to assess social skills 
and problem behaviour among children. SSRS draws on multiple sources of information and questionnaires 
were completed by children themselves, parents and teachers not involved in ART trainings. 

Social Skills Rating System (SSRS) (Gresham & Elliot, 1990) (assessment by teachers/staff members 
and self-report). The Social Skills Rating System allows for acquiring a more complete picture of social 
behaviours from teachers, parents, and even students themselves. Items of the child’s version of the 
questionnaire are combined in four scales - Cooperation, Assertion, Empathy and Self-control. Answer 
options vary from l=never to 4=very often (Ogden, 2003). 

The parental and teacher versions of the SSRS assess the domains of social skills, problem behaviour, 
and academic competence. The Social Skills Scale has five subscales: Cooperation, Assertion, Responsibility, 
Empathy, and Self-Control. The Problem Behaviours Scale has three subscales that measure Externalizing 
Problems and Internalizing Problems. 

The reliability and validity of the subscales of social competence and behavioural problems have been 
assessed in several studies (Demary, Maray, Ruffalo, et al., 1995; Gresham & Elliot, 1990), supporting the 
use of the instrument in research settings. In our study, internal consistency coefficients for the four scales of 
the children’s version of the questionnaire ranged from acceptable to good, including 0.86 for Cooperation, 
0.82 for Assertion, 0.83 for Empathy and 0.66 for Self-control, 

Internal consistency coefficients for the seven scales of the parental version of the questionnaire 
ranged from acceptable to good, including 0.82 for Cooperation, 0.86 for Assertion, 0.85 for Responsibility, 
0.80 for Self-control, 0.89 for Externalizing Problems and 0.65 for Internalizing Problems. 

Internal consistency coefficients for the six scales of the teachers’ version of the questionnaire ranged 
from acceptable to good, including 0.89 for Cooperation, 0.84 for Assertion, 0.87 for Self-control, 0.86 for 
Externalizing Problems and 0.62 for Internalizing Problems. 

The translations of the questionnaire into Russian followed established guidelines, including 
appropriate use of independent back-translations (Sartorius & Kuyken, 1994). Finally, an official interpreter 
made independent back-translations. The versions obtained were compared with originals, and inconsistencies 
were analyzed and corrected. 
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To ensure the validity of the collected data, institutions and schools which participated in the study 
obtained recommendations about randomized allocation of participants to intervention and control groups. 
Children, parents and teachers completed the SSRS questionnaire at the institution or at school while parents 


were also given the opportunity to do it at home. Institution and school staff involved in completing the 


questionnaires were different from those who were trained in ART. Completion of the questionnaire was 
conducted one week prior to a start of the ART implementation (pre-test). Post-test was performed within one 
week after the ART intervention. 

Results and discussion 

Children: Overall effects. An overall prediction for the SSRS pre vs. post scores was that the scores 
should increase in the ART group and remain relatively stable in the control group. We therefore subjected 
the overall pre and post SSRS scores to repeated measures ANOVA with intervention (ART vs. control) as 
between group factors and the pre-post scores as the repeated factor. The specific predictions were tested by 
contrast analyses (Rosenthal & Rosnow, 1985). The first contrast (i.e. pre vs. post scores within the ART 
group) indicated a significant effect, F (1, 208) = 4.75, p < .05. The second contrast (i.e. pre vs. post scores 
within the control group) was not significant, F (1, 208) = 1.34, p = .25. This indicates that the ART 
intervention was associated with a significant positive change, and no change in the control group. These 
results are shown in Figure 1. 
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Figure 1. Change in overall SSRS competence scores between PRE and POST in the ART and 


control groups (students’ self-report data). 


Because intervention effects are often modulated by personal characteristics (age, sex) and factors 
related to the intervention (implementation quality that may differ between institutions), we subjected the 
effect data to an overall ANCOVA (GLM) with these factors as discrete (intervention, sex) and continuous 
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(age, institution) predictor variables, and change in pre vs. post competence scores as the dependent variable. 
The analysis indicated no significant effect of the intervention, F (1.208) = 3.52, p = .062. Other studied 
factors also indicated no significant effects. 

Because age level previously has been shown to be related to intervention efficacy (Langeveld, 
Gundersen, & Svartdal, 2012), when analyzing the data we arranged the participants in three age groups, 6-9, 
10-14, and 15 years and older to compare pre-post changes on social competence. This arrangement was 
made according to age related class level in the Russian school system where age 6-9 is related to primary 
school, 10-14 middle school and 15 and older to secondary school. An ANOVA with intervention and age 
levels as predictor variables indicated that overall social competence levels decreased significantly with 
increasing age, F (2, 207) = 3.19, p < .05. Further, as is shown in Figure 2, reliable effects of the ART 
intervention seemed to occur in the two lower age groups, but not in the oldest group. Note, however, that the 
number of participants in the 15+ age group was low (8 in the ART group, 12 in the control group). Contrast 
analysis of the predicted pre vs. post change in the 6-8 years ART group was significant, F (1, 237) = 5.34, p 
< .025. The corresponding contrast in the 10-14 years ART group was not significant, F (1, 237) = 2.98, p < 
. 10 . 



Figure 2. Change in overall SSRS competence scores between pre and post-test in the ART and 
control groups grouped according to respondents’ age levels (students’ self-report data). 

Children: Subscale analyses. To examine the nature of the SSRS score changes between pre and post 
indicated in the two lower age groups in Figure 2, we plotted the subscale scores (i.e. cooperation, assertion, 
empathy and self-control) for the ART vs. control groups. As indicated in Figure 3, the younger participants 
in the ART group demonstrated increased levels of corresponding magnitude on all four subscales from pre to 
post, whereas no changes were observed in the control group. 
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Figure 3. Changes in pre vs. post-test scores on the SSRS subscales, ART and control groups 

(students’ self-report data). 

Parents and teachers: Social competence. The parents’ and teachers’ SSRS social competence scores 
were subjected to repeated measures ANOVA with intervention (ART vs. control) as the between-group 
factor and time (pre vs. post) as the repeated measures factor. The ANOVA indicated a significant effect of 
time, F (1, 64) = 58.16, p < .001, reflecting that SSRS scores for both informants and both conditions changed 
markedly between pre and post (Figure 4). Obviously, there were no differences between the ART and control 
groups. 
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Figure 4. Changes in pre vs. post-test social competence scores according to parents and 

teachers 
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Parents and teachers: Problem scores. The parents’ and teachers’ SSRS external problem scores 
were analyzed by repeated measures ANOVA with intervention (ART vs. control) as the between-group 
factor and time (pre vs. post) as the repeated measures factor. The ANOVA indicated a significant effect of 
time, F (1, 59) = 10.87, p = .005, indicating a general reduction in problem scores from pre- to post 
assessment. No other significant effects were observed (Figure 5). 




Figure 5. Changes in pre vs. post-test problem scores according to parents and teachers 


Discussion 

The study aimed to examine the efficacy of the ART programme with regards to pro-social skill 
acquisition and decrease in problem behaviour. Based on the children’s self-reports, the results indicated 
positive changes from pre to post in overall SSRS competence scores among children from the ART group. 
The ART group demonstrated increased levels on all four SSRS subscales, namely cooperation, assertion, 
empathy and self-control. In contrast, pre- and post-test comparisons of overall SSRS competence scores and 
SSRS subscales, revealed no change in the control group. These results are in line with previous studies 
conducted in Norway and USA on the efficacy of the ART programme (Gundersen & Svartdal, 2006; Nugent 
& Bruley, 1998). 

When examining the role of age as a moderator of the efficacy of the programme, we found that 
younger participants demonstrated greater improvement in social training. Indeed, reliable effects of the ART 
intervention were found among children 14-years old and younger, with no changes observed among children 
older than 15. This finding is in line with previous studies (e.g., Langeveld, Gundersen & Svartdal, 2012) that 
older children often benefit less from competence training compared to younger children. In the age group 
>15, a positive change was found in the control group, with no change in the ART group. We attribute this to 
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the low number of pupils in the oldest group and hence the outcome in this group may be unreliable. It is also 
possible that the ART intervention is contra-productive for this age group. Another study involving more 
children from this age group is needed to confirm or reject this conclusion. 

When parent and teacher pre- and post-ratings were analysed, it was found that children from the 
intervention and control groups demonstrated similar positive changes in SSRS scores. Also, children from 
intervention and control groups demonstrated less externalising problems in the pre- and post-comparisons. 
One of the possible explanations for this is related to the day-to-day interaction between the children at the 
institution and the transfer of positive changes in behaviour from ART children to those in the control group. 
This effect was noted earlier in several studies (Gundersen & Svartdal, 2010; Kazdin, 1998) and may be 
attributed to a diffusion of treatment. Thus, measures intended for the intervention group may have affected 
the control group. A more subtle form of diffusion of treatment is the possibility that changed behaviour in 
the intervention group may have affected behaviour in the control group, thus representing some form of 
indirect diffusion of treatment (Gundersen & Svartdal, 2010). In our study, children from control and 
intervention groups had close daily contact and it is likely that those receiving the ART programme applied 
their new skills outside the training room. Although this represents a problem by threatening the internal 
validity of this study, it also indicates a potential positive side-effect of ART programme implementations in 
situations with a high degree of interaction between participants. 

Although, the overall result of the evaluation of the ART programme is positive, several limitations of 
the study must be mentioned. Although randomization was carried out by the teaching staff following a strict 
protocol, there might still have been a tendency for the teaching staff to allocate those pupils with the highest 
level of difficulty to intervention groups. Secondly, often the same teacher completed the teacher’s reports 
for several children, which could influence the estimation accuracy when providing the assessment. The 
involvement of more teachers in research can potentially provide better assessment of children and improve 
the quality of data. Thirdly, the achievement of better results is guaranteed when the programme is culturally 
sensitive and adapted accordingly. It is common for practitioners to change or adapt evidence-based 
programmes as they implement them, whether intentionally or not. The adaption, thus, must assume meeting 
local needs without compromising the efficacy of the programme. Fourth, the efficacy of the programme is 
dependent on whether the implementation is systematic and well-focused. To our knowledge, most of the 
ART trainers had extensive theoretical and practical training before using the programme and this could 
potentially guarantee the fidelity of the implementation. Nonetheless, it was not possible to prevent cases 
when the implementation was influenced by, for example, illness of trainers or participants, children’s refusal 
to continue with ART or fill out the questionnaires, and overload of trainers with other work duties at their 
institution. Better results from the implementation can be achieved when the intervention is fully integrated 
not only into the institution’s approved intervention plan but also into the institution’s daily routines. This 
involves quality systems that secure proper generalization into the ART group’s natural environment as well 
as regular supervision and sufficient time for trainers to prepare and debrief each session. Olsen and 
Gundersen (2012) have developed an assessment scale measuring the degree of proper implementation for use 
in future research in ART. Such measures related to implementation quality would also allow for a focused 
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examination of the relationship between implementation quality and outcome efficacy.In conclusion, our 
study revealed significant positive changes in skills among children who received the ART programme. The 
study also found that many children from both intervention and control groups demonstrated similar 
tendencies in decreased behavioural problems and we attributed this outcome to diffusion of treatment effect. 

As far as we know, this is the first evaluation study on efficacy of the ART programme in Russia. The 
results suggest initial support for the use of the programme as an effective method for increasing social skills 
and decreasing behavioural problems among children from social institutions and schools. Future studies 
should investigate whether the ART programme is also effective when it is implemented among Russian 
children and adolescents at risk of offending. 
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